13 research outputs found
Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking
Extraction from raw text to a knowledge base of entities and fine-grained
types is often cast as prediction into a flat set of entity and type labels,
neglecting the rich hierarchies over types and entities contained in curated
ontologies. Previous attempts to incorporate hierarchical structure have
yielded little benefit and are restricted to shallow ontologies. This paper
presents new methods using real and complex bilinear mappings for integrating
hierarchical information, yielding substantial improvement over flat
predictions in entity linking and fine-grained entity typing, and achieving new
state-of-the-art results for end-to-end models on the benchmark FIGER dataset.
We also present two new human-annotated datasets containing wide and deep
hierarchies which we will release to the community to encourage further
research in this direction: MedMentions, a collection of PubMed abstracts in
which 246k mentions have been mapped to the massive UMLS ontology; and TypeNet,
which aligns Freebase types with the WordNet hierarchy to obtain nearly 2k
entity types. In experiments on all three datasets we show substantial gains
from hierarchy-aware training.Comment: ACL 201
Pseudointelligence: A Unifying Framework for Language Model Evaluation
With large language models surpassing human performance on an increasing
number of benchmarks, we must take a principled approach for targeted
evaluation of model capabilities. Inspired by pseudorandomness, we propose
pseudointelligence, which captures the maxim that "(perceived) intelligence
lies in the eye of the beholder". That is, that claims of intelligence are
meaningful only when their evaluator is taken into account. Concretely, we
propose a complexity-theoretic framework of model evaluation cast as a dynamic
interaction between a model and a learned evaluator. We demonstrate that this
framework can be used to reason about two case studies in language model
evaluation, as well as analyze existing evaluation methods.Comment: EMNLP 2023 Finding
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
Recursion is a prominent feature of human language, and fundamentally
challenging for self-attention due to the lack of an explicit recursive-state
tracking mechanism. Consequently, Transformer language models poorly capture
long-tail recursive structure and exhibit sample-inefficient syntactic
generalization. This work introduces Pushdown Layers, a new self-attention
layer that models recursive state via a stack tape that tracks estimated depths
of every token in an incremental parse of the observed prefix. Transformer LMs
with Pushdown Layers are syntactic language models that autoregressively and
synchronously update this stack tape as they predict new tokens, in turn using
the stack tape to softly modulate attention over tokens -- for instance,
learning to "skip" over closed constituents. When trained on a corpus of
strings annotated with silver constituency parses, Transformers equipped with
Pushdown Layers achieve dramatically better and 3-5x more sample-efficient
syntactic generalization, while maintaining similar perplexities. Pushdown
Layers are a drop-in replacement for standard self-attention. We illustrate
this by finetuning GPT2-medium with Pushdown Layers on an automatically parsed
WikiText-103, leading to improvements on several GLUE text classification
tasks.Comment: Accepted at EMNLP 2023 (Long Papers
Grokking of Hierarchical Structure in Vanilla Transformers
For humans, language production and comprehension is sensitive to the
hierarchical structure of sentences. In natural language processing, past work
has questioned how effectively neural sequence models like transformers capture
this hierarchical structure when generalizing to structurally novel inputs. We
show that transformer language models can learn to generalize hierarchically
after training for extremely long periods -- far beyond the point when
in-domain accuracy has saturated. We call this phenomenon \emph{structural
grokking}. On multiple datasets, structural grokking exhibits inverted U-shaped
scaling in model depth: intermediate-depth models generalize better than both
very deep and very shallow transformers. When analyzing the relationship
between model-internal properties and grokking, we find that optimal depth for
grokking can be identified using the tree-structuredness metric of
\citet{murty2023projections}. Overall, our work provides strong evidence that,
with extended training, vanilla transformers discover and use hierarchical
structure.Comment: ACL 202